The present invention relates to a method and an apparatus of image processing, and a data storage medium and, more particularly to a method and an apparatus which are capable of coding or decoding an image signal without damaging a regenerated video even if a system which processes an image signal has low processing ability, and a data storage medium which contains a program for implementing coding and decoding the image signal by the method and the apparatus by software.
In a prior art coding method for a digital moving image, since temporal correlation, i.e., correlation of image information between frames, is high in the moving image, a difference value between an image signal of a target frame to be coded and a prediction signal obtained from an image signal of a previous frame which has been coded (before the target frame is coded) is coded, in order to eliminate redundant image information between frames and thereby reduce temporal redundancy.
This coding process is called a xe2x80x9cdifferential coding processxe2x80x9d,and is performed for each of unit areas into which one frame is divided. The frame corresponds to one frame according to MPEG (Moving Picture Experts Group) 1 and 2, and corresponds to an object area, i.e., an image display space corresponding to an object image according to MPEG4.
To be specific, the differential coding process detects a prediction area in the previous frame which comprises an image signal having the smallest error with respect to an image signal (input image signal) of a target unit area to be coded, by motion prediction using the image information of the previous frame, and performs coding to a difference signal corresponding to a difference value between the image signal of the target unit area and the prediction signal (an image signal in the prediction area) for the target unit area along with prediction relevant information associated with the motion prediction. The prediction relevant information is, for example, information indicating a position of the prediction area, namely, a motion vector.
Meanwhile, a differential decoding process regenerates an image signal based on a coded difference signal generated by the differential coding process. The differential decoding process uses motion compensation to predict a regenerated image signal (decoded image signal) for a target unit area to be decoded and thereby generate a prediction signal (decoded prediction image signal) according to the prediction relevant information, decodes the coded difference signal of the target unit area to generate a decoded difference signal, and adds the prediction signal to the decoded difference signal to generate a regenerated image signal corresponding to the target unit area.
As described above, in the conventional process for coding or decoding the image signal, techniques for compressing image data such as motion prediction or motion compensation have been employed. A specific example of implementation of these is data compression according to MPEG1 or MPEG2 as an international standard.
Attempts are being made to extend standardization of techniques according to MPEG4 being standardized, by introducing new motion prediction or motion compensation into them, in addition to the motion prediction and motion compensation according to MPEG1 or MPEG2.
The motion prediction detects the information indicating the position of the prediction area, i.e., the motion vector, in the coding process. The motion compensation generates the prediction signal according to the motion vector in the coding process or the decoding process.
One of such introduction is overlapping motion compensation. Hereinafter, a description will be given of normal motion compensation and overlapping motion compensation by an image processing technique according to MPEG1 and MPEG2. For the sake of simplicity, a luminance signal is shown as a signal to be subjected to motion compensation. The overlapping motion compensation is commonly performed in the decoding process, including a local decoding process in the coding process, and therefore the decoding process will be discussed.
In a prior art decoding process according to MPEG1 and MPEG2, the motion compensation is performed for each unit area corresponding to a two-dimensional image space (macroblock) comprising 256 pixels (16xc3x9716 pixels).
FIGS. 7(a)-7(d) show motion compensation in the decoding process.
Turning now to FIG. 7(a), for a target macroblock MB(n) in a target frame F(n) being decoded at a current time t(n), a motion vector MV(n) is used to find a prediction area PR(n) in a previous frame F(n-1) which has been decoded at a past time t(n-1) The target macroblock MB(n) and the corresponding macroblock MB(n-1) are present in the same position in the target frame F(n) and the previous frame F(n-1), respectively. The prediction area PR(n) is apart from the corresponding macroblock MB(n-1) by a displacement indicated by the motion vector MV(n), in the previous frame F(n-1)
Then, an image signal included in the prediction area PR(n) is found as a prediction signal (decoded prediction image signal) for a decoded image signal of the target macroblock MB(n).
FIGS. 7(b)), 7(c), and (d), show the macroblock MB(n-1), the prediction area PR(n), and the target macroblock MB(n) in FIG. 7(a), respectively.
The overlapping motion compensation is performed for each unit area, i.e., a two-dimensional image space (block) comprising 64 pixels (8xc3x978 pixels). 4 of these blocks correspond to one macroblock. In other words, one macroblock is composed of (2xc3x972) blocks, as shown in FIG. 8.
In the overlapping motion compensation, the motion vector is present for each block or for each macroblock. Assume that motion vectors for the 4 blocks in the macroblock are the same when the motion vector is for the macroblock.
A description will be given of overlapping motion compensation with reference to FIG. 9.
Herein, a description will be given of generation of a prediction signal (a decoded prediction image signal) of the target block EC(n) in the target frame F(n).
Initially, by the use of the motion vector MVC of the target block BC(n), motion vectors MVU, MVD, MVR, and MVL of neighboring blocks BU(n), BD(n), BR(n), and BL(n) which are positioned above, below, right, and left with respect to the block BC(n), 5 prediction areas of the target block BC(n) are found by using the corresponding block BC(n-1) in the previous frame F(n-1) as a criterion. The motion vectors MVU, MVD, MVR, and MVL indicate areas PU(n), PD(n), PR(n), and PL(n) in the previous frame F(n-1) as the prediction areas of the blocks BU(n), BD(n), BR(n), and BL(n) in the previous frame F(n-1), and therefore the 4 prediction areas of the target block BC(n) found by the use of these motion vectors are, as shown in FIG. 10(a), areas PU(n)xe2x80x2, PD(n)xe2x80x2, PR(n)xe2x80x2, and PL(n)xe2x80x2. In addition, by the use of the motion vector MVC of the target block BC(n), the prediction area PC(n) is found for the target block BC(n).
Subsequently, from pixels in the 5 prediction areas PC(n), PU(n)xe2x80x2, PD(n)xe2x80x2, PR(n)xe2x80x2 and PL(n)xe2x80x2, prediction images PGC, PGU, PGD, PGR, and PGL in FIG. 10(b) are found. It should be remembered that unnecessary pixels in respective prediction areas are not used. To be specific, the prediction images of the prediction area PU(n)xe2x80x2, the prediction area PD(n)xe2x80x2, the prediction area PR(n)xe2x80x2, and the prediction area PL(n)xe2x80x2 are formed from pixels in the upper portion, pixels in the lower portion, pixels in the right portion, and pixels in the left portion, respectively.
Then, values of pixels in the prediction images, PGC, PGU, PGD, PGR, and PGL, are weighted by values shown by weighting matrixes WVC, WVU, WVD, WVR, and WVL(see FIG. 10(c)). Weighting values shown in respective weighting matrixes are added for each pixel, to obtain xe2x80x9c8xe2x80x9d. Pixel values Pg of a composite image PG is found according to the following expression:
Pq=(Pqcxc2x7Wvc+Pquxc2x7Wvu+Pgdxe2x88x92Wvd+Pgrxc2x7Wvr+Pg1xc2x7Wv1)/8
wherein Pgc, Pgu, Pgd, Pgr, and Pg1 are pixel values of the respective prediction images PGC, PGU, PGD, PGR, and PGL, and Wvc, Wvu, Wvd, Wvr, and Wv1 are weighting values shown by the respective weighting matrixes.
The composite image PG resulting from this composition is the prediction image resulting from the overlapping motion compensation, and an image signal corresponding thereto is the prediction signal (decoded prediction image signal).
In the overlapping motion compensation, when the target block to-be-subjected to motion compensation is positioned in the lower portion of a macroblock, the motion vector of the target block is used as motion vectors of neighboring blocks in the lower portion, since the macroblock including the neighboring blocks has not been decoded. Needless to say, the overlapping motion compensation is performed in local decoding process included in coding process in the same manner that the overlapping motion compensation is performed in the decoding process.
As described above, the normal motion compensation generates the prediction image (prediction signal) by the use of the motion vector of the area to-be-subjected to motion compensation (macroblock), while the overlapping motion compensation generates the prediction image (prediction signal) by the use of the motion vectors of the neighboring blocks adjacent to the target block as well as the motion vector of the target block to-be-subjected to the overlapping motion compensation. For this reason, in contrast with the normal motion compensation, the overlapping motion compensation makes difference values between the image signal of the target block (values of respective pixels) and the prediction signal (prediction values of respective pixels) uniform in a block. This means that large coding distortion which is locally present in the macroblock is distributed therein. The overlapping motion compensation improves visual image quality, but requires more calculation than the normal motion compensation.
A description will now be given of an object-by-object motion compensation in the decoding process according to MPEG4.
In image display according to MPEG4, objects areas in a target frame and a previous frame are present in different positions.
Turning now to FIG. 12, at a decoding end, for a target macroblock MB(n) in an object area OR(n) in a target frame F(n) being decoded at a current time t(n), a motion vector MV(n) is used to find a prediction area PR(n) in the corresponding object area OR(n-1) in a previous frame F(n-1) which has been decoded at a past time t(n-1).
The prediction area PR(n) is positioned apart from the corresponding macroblock MB(n-1) by a displacement indicated by the motion vector MV(n), in the previous frame F(n-1).
While in the previous frame P(n-1), a position P(n) in the object area OR(n) at t(n) differs from a position P(n-1) in an object area OR(n-1) at t(n-1), a position Pmb(n) of the macroblock MB(n) in the object area OR(n) in the target frame F(n), a position Pmb(n-1) of the macroblock MB(n-1) in-the object area OR(n-1) in the previous frame F(n-1), and a position Ppr (n) of the prediction area PR(n) are determined as absolute positions with respect to a position PO on a upper-left corner of each frame.
The image signal of the prediction area PR(n) is found as the prediction signal (decoded prediction image signal) for the decoded image signal of the target macroblock MB(n) in the target frame F(n).
As in the case of MPEG1 and 2, the overlapping motion compensation according to MPEG4 is performed for each unit area corresponding to a two-dimensional image space (block) comprising 64 pixels (8xc3x978 pixels), for the object areas OR(n) and OR(n-1) as shown in FIG. 12, as already shown in FIG. 9.
Thus, to improve coding efficiency in the coding process, various prediction signal generating methods have been proposed and adopted by standards for the coding method, and more and more methods would be expected to be conceived.
However, in many cases, while new motion prediction or new motion compensation improves coding efficiency, it makes calculation more complicated than the conventional process, as already described in the overlapping motion compensation. Accordingly, high processing ability is demanded of an image processing system for real time processing.
In addition, since a system which performs the prior art image coding process and image decoding process, performs motion compensation by calculation according to a prediction signal generating method decided in the coding process, frames are sometimes skipped in a video regenerated in real time if processing ability in the decoding process is low.
In other words, in a case where the prior art image processing system has low processing ability resulting from introduction of new motion prediction or motion compensation, quality of a regenerated image is adversely affected.
It is an object of the present invention to provide a method and an apparatus which are capable of performing switching of a process for generating a prediction signal for a target image signal between a lightly loaded process and a heavily loaded process, depending upon the calculation load, when coding or decoding an image signal, thereby performing motion prediction or motion compensation with high efficiency while suppressing adverse effects on a regenerated video, and a data storage medium which contains a program for implementing image processing by the method and the apparatus in a computer system.
Other objects and advantages of the invention will become apparent from the detailed description that follows. The detailed description and specific embodiments described are provided only for illustration since various additions and modifications within the spirit and scope of the invention will be apparent to those skill in the art from the detailed description.
According to a first aspect of the present invention, an image processing method for coding an input image signal for each of unit areas into which a frame is divided and outputting a coded image signal, and generating a prediction image signal for a target unit area to be coded in a desired unit area by calculation according to prediction based on a decoded image signal obtained by decoding the coded image signal, comprises: detecting largeness of calculation load in the process for coding the input image signal, and performing switching of the prediction in the process for generating the prediction image signal according to the detected largeness of the calculation load. Therefore, the calculation load does not exceed the processing ability of the system which performs the coding process. Thereby, at the decoding end, frames are not skipped in the video regenerated in real time. As a result, the image quality of the regenerated image is not adversely affected by the increased load resulting from the process for generating the prediction signal.
According to a second aspect of the present invention, in the image processing method of the first aspect, the prediction in the process for generating the prediction image signal is switched between a first prediction which requires a large amount of calculation and a second prediction which requires a small amount of calculation, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Thereby, coding efficiency is improved without degrading the image quality of the regenerated image.
According to a third aspect of the present invention, in the image processing method of the first aspect, the prediction adapted to the detected largeness of the calculation load is selected among plural types of prediction which differ from one another in amount of calculation, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Therefore, it is possible to select the amount of calculation among many alternatives within a wider range. In addition, since the proportion of the amount of signals processed in the motion prediction to:the total amount of signals processed in the coding process is high, the change of the motion prediction significantly reduces the calculation load in the transmission system.
According to a fourth aspect of the present invention, in the image processing method of the first aspect, the process for generating the prediction image signal includes motion prediction for finding a motion vector indicating a position of a prediction area of the target unit area by predetermined pixel precision, and motion compensation for finding an image signal of the prediction area as the prediction image signal by the use of the motion vector, the motion prediction is switched between motion prediction which requires a large amount of calculation and motion prediction which requires a small amount of calculation, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load, and the motion compensation is switched between motion compensation which requires a large amount of calculation and motion compensation which requires a small amount of calculation, according to the result of comparison. Therefore, it is possible to select the amount of calculation among many alternatives within a wider range.
According to a fifth aspect of the present invention, in the image processing method of the first aspect, the process for generating the prediction image signal includes motion prediction for finding a motion vector indicating a position of a prediction area of the target unit area by predetermined pixel precision, and motion compensation for finding an image signal of the prediction area as the prediction image signal by the use of the motion vector, and the motion prediction is switched between motion prediction for finding a motion vector of higher pixel precision and motion prediction for finding a motion vector of lower pixel precision, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Therefore, the relatively simple process, i.e., the change of the pixel precision, can improve coding efficiency without degrading the regenerated image.
According to a sixth aspect of the present invention, in the image processing method of the first aspect, the process for generating the prediction signal includes motion prediction for finding a motion vector indicating a position of a prediction area of the target unit area by predetermined pixel precision, and motion compensation for finding an image signal of the prediction area as the prediction image signal by the use of the motion vector, and the motion compensation is switched between motion compensation for finding the prediction image signal by the use of a motion vector of higher pixel precision and motion compensation for finding the prediction image signal by the use of a motion vector of lower pixel precision, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Therefore, the relatively simple process can improve coding efficiency without degrading the regenerated image.
According to a seventh aspect of the present invention, in the image processing method of the sixth aspect, the motion compensation performed by the use of the motion vector of higher pixel precision is switched between first motion compensation which is heavily loaded and second motion compensation which is lightly loaded, according to the result of comparison, and the motion compensation performed by the use of the motion vector of lower pixel precision is switched between first motion compensation which is heavily loaded and second motion compensation which is lightly loaded, according to the result of comparison. Therefore, it is possible to select the amount of calculation among many alternatives within a wide range.
According to an eighth aspect of the present invention, in the image processing method of the first aspect, the process for generating the prediction image signal includes motion prediction for finding a motion vector indicating a position of a prediction area of the target unit area by predetermined pixel precision, and motion compensation for finding an image signal of the prediction area as the prediction image signal by the use of the motion vector, and one of plural types of motion compensation which use motion vectors of different pixel precision is selected as the motion compensation, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Therefore, the relatively simple process can change amount of calculation appropriately.
According to a ninth aspect of the present invention, an image processing method for decoding an input coded image signal for each of unit areas into which a frame is divided and outputting a decoded image signal, and generating a decoded prediction image signal for a target unit area to be decoded in a desired unit area by calculation according to predetermined motion compensation based on the decoded image signal, comprises detecting largeness of calculation load in the process for decoding the coded image signal, and performing switching of motion compensation in the process for generating the prediction image signal according to the detected largeness of the calculation load. Therefore, the image quality of the regenerated image is not adversely affected by the increased load resulting from the process for generating the prediction signal.
According to a tenth aspect of the present invention, in the image processing method of the ninth aspect, the motion compensation is switched between first motion compensation which requires large amount of calculation and second motion compensation which requires small amount of calculation, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load.
Therefore, it is possible to perform decoding process adapted to the coding process which improves coding efficiency without degrading the image quality of the regenerated image.
According to an eleventh aspect of the present invention, in the image processing method of the ninth aspect, the motion compensation adapted to the detected largeness of the calculation load is selected among plural types of motion compensation which differ from one another in amount of calculation, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Therefore, it is possible to select amount of calculation among many alternatives within a wide range.
According to a twelfth aspect of the present invention, in the image processing method of the ninth aspect, the motion compensation is switched between motion compensation for finding the decoded prediction image signal by the use of a motion vector of higher pixel precision and motion compensation for finding the decoded prediction image signal by the use of a motion vector of lower pixel precision, according to a result of comparison between a threshold of the calculation load determined from processing ability required for the process for generating the prediction signal in real time and the detected largeness of the calculation load. Therefore, the relatively simple process can change amount of calculation appropriately.
According to a thirteenth aspect of the present invention, in the image processing method of the twelfth aspect, the motion compensation performed by the use of the motion vector of higher pixel precision is switched between first motion compensation which is heavily loaded and second motion compensation which is lightly loaded, according to the result of comparison, and the motion compensation performed by the use of the motion vector of lower pixel precision is switched between first motion compensation which is heavily loaded and second motion compensation which is lightly loaded, according to the result of comparison. Therefore, Therefore, it is possible to select the amount of calculation among many alternatives within a wider range
According to a fourteenth aspect of the present invention, an image processing apparatus including a coding unit for coding an input image signal for each of unit areas into which a frame is divided and outputting a coded image signal, and a predicting unit for generating a prediction image signal for a target unit area to be coded in a desired unit area by calculation according to predetermined prediction, wherein the predicting unit includes calculation load detecting means for detecting largeness of calculation load in the process for coding the input image signal, and performs switching of the prediction in the process for generating the prediction signal according to the detected largeness of the calculation load. Therefore, the calculation load does not exceed the processing ability of the system which performs the coding process. Thereby, at the decoding end, frames are not skipped in the video regenerated in real time. As a result, the image quality of the regenerated image is not adversely affected by the increased load resulting from the process for generating the prediction signal.
According to a fifteenth aspect of the present invention, in the image processing apparatus of the fourteenth aspect, the predicting unit includes a motion detector which outputs a motion vector indicating a position of a prediction area in a previous frame comprising a prediction image signal for the target unit area, based on the input image signal of the target unit area in a target frame to be coded and a decoded image signal of the previous frame which has been decoded; and a motion compensator for generating the prediction image signal by the use of the motion vector of the target unit area which is output from the motion detector; the motion detector outputting one of a first motion vector of lower pixel precision and a second motion vector of higher pixel precision to the motion compensator as the motion vector, according to the detected largeness of the calculation load. Thereby, coding efficiency is improved without degrading the image quality of the regenerated image.
According to a sixteenth aspect of the present invention, an image processing apparatus including a decoding unit for decoding an input coded image signal for each of unit areas into which a frame is divided and outputting a decoded image signal, and a predicting unit for generating a decoded prediction image signal for a target unit area to be decoded in a desired unit area by calculation according to predetermined motion compensation, wherein the predicting unit includes calculation load detecting means for detecting largeness of calculation load in the process for decoding the coded image signal, and performs switching of motion compensation in the process for generating the decoded prediction image signal, according to the detected largeness of the calculation load. Therefore, the calculation load does not exceed the processing ability of the system which performs the coding process. Thereby, at the decoding end, frames are not skipped in the video regenerated in real time. As a result, the image quality of the regenerated image is not adversely affected by the increased load resulting from the process for generating the prediction signal.
According to a seventeenth aspect of the present invention, in the image processing apparatus of the sixteenth aspect, the predicting unit generates the decoded prediction image signal by the use of a motion vector indicating a position of a prediction area in a previous frame which has been decoded before the target frame is decoded and comprises the decoded prediction image signal for the target unit area, and includes a motion compensator which selects and performs one of first motion compensation which requires small amount of calculation for generating the decoded prediction image signal for the target unit area by the use of the motion vector of the target unit area and second motion compensation which requires large amount of calculation for generating the decoded prediction image signal by the use of the motion vector of the target unit area and its neighboring unit areas, according to the detected largeness of calculation load. Therefore, it is possible to perform decoding process adapted to the coding process which improves coding efficiency without degrading the image quality of the regenerated image.
According to an eighteenth aspect of the present invention, a data storage medium for storing a program which makes computer image coding process, comprises coding an input image signal for each of unit areas into which a frame is divided and outputting a coded image signal, and generating a prediction image signal for a target unit area to be coded in a desired unit area by calculation according to predetermined prediction based on a decoded image signal obtained by decoding the coded image signal, wherein largeness of calculation load in the process for coding the input image signal is detected, and switching of prediction in the process for generating the prediction image signal is performed according to the detected largeness of the calculation load. Therefore, the computer can realize the coding process wherein the calculation load does not exceed the processing ability of the system which performs the coding process, and thereby at the decoding end, frames are not skipped in the video regenerated in real time, whereby the image quality of the regenerated image is not adversely affected by the increased load resulting from the process for generating the prediction signal.
According to a nineteenth aspect of the present invention, a data storage medium for storing a program which makes a computer perform image decoding process, comprises the step of decoding an input coded image signal for each of unit areas into which a frame is divided and outputting a decoded image signal, and generating a decoded prediction image signal for a target unit area to be decoded in a desired unit area by calculation according to predetermined motion compensation based on the decoded image signal, wherein largeness of calculation load in the process for decoding the coded image signal is detected, and switching of motion compensation in the process for generating the decoded prediction image signal is performed according to the detected largeness of the calculation load. Therefore, the computer can realize the decoding process wherein the calculation load does not exceed the processing ability of the system which performs the decoding process, and thereby at the decoding end, frames are not skipped in the video regenerated in real time.
FIG. 1 is a block diagram showing an image coding apparatus according to a first embodiment of the present invention.
FIGS. 2(a) and 2(b) are flowcharts, wherein FIG. 2(a) shows a coding process performed by the image coding apparatus of the first embodiment, and FIG. 2(b) shows a process for generating a prediction signal.
FIGS. 3(a) and 3(b) are flowcharts showing processes for generating prediction signals performed by image coding apparatuses according to first and second modifications of the first embodiment, respectively.
FIG. 4 is a block diagram showing an image decoding apparatus according to a second embodiment of the present invention.
FIGS. 5(a) and 5(b) are flowcharts showing a decoding process and a motion compensation process performed by the image decoding apparatus of the second embodiment, respectively.
FIGS. 6(a)-6(c) are flowcharts showing motion compensation processes performed by image decoding apparatuses according to first, second, and third modifications of the second embodiment, respectively.
FIGS. 7(a)-7(d) are diagrams for explaining a motion compensation process performed by a prior art image decoding apparatus according to MPEG 1 and 2.
FIG. 8 is a diagram showing relationship between a macroblock comprising (16xc3x9716) pixels and a block comprising (8xc3x978) pixels.
FIG. 9 is a diagram for explaining an overlapping motion compensation process in a prior art image decoding method.
FIGS. 10(a)-10(c) are diagrams for explaining a process for generating a prediction signal in the overlapping motion compensation process.
FIGS. 11(a)-11(c) are diagrams for explaining a data storage medium which contains a program for implementing a coding process or a decoding process performed by the image processing apparatus of each of the above embodiments in a computer system.
FIG. 12 is a diagram for explaining a motion compensation process in the prior art image decoding method according to MPEG4.